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Recovery of Signals with Low Density 


Christoph Studer 


Abstract 

Sparse signals (i.e., vectors with a small number of non-zero entries) build the foundation of most 
kernel (or nullspace) results, uncertainty relations, and recovery guarantees in the sparse signal-processing 
and compressive-sensing literature. In this paper, we introduce a novel signal-density measure that extends 
the common notion of sparsity to non-sparse signals whose entries’ magnitudes decay rapidly. By taking 
into account such magnitude information, we derive a more general and less restrictive kernel result 
and uncertainty relation. Furthermore, we demonstrate the effectiveness of the proposed signal-density 
measure by showing that orthogonal matching pursuit (OMP) provably recovers low-density signals with 
up to 2 X more non-zero coefficients compared to that guaranteed by standard results for sparse signals. 


I. Introduction 

In this paper, we consider kernel (or nullspace) results providing conditions for which Ax / Omxi 
and uncertainty relations for pairs of signals (vectors) x G and z G that satisfy Ax = Bz, 
where A G and B G are dictionaries, i.e., matrices whose columns are normalized to 

unit ^ 2 -norm. We furthermore study conditions for which orthogonal matching pursuit (OMP) recovers 
signals x from an underdetermined system of linear equations y = Ax with M < Na- 

A. Contributions and Paper Outline 

In contrast to existing kernel results, uncertainty relations, and recovery guarantees that have been 
derived for sparse signals, i.e., for vectors with only a small number of nonzero entries, ifTl- lfm . the 
results developed in this paper make use of the following definition (see Section |II] for the details). 

Definition 1 (5-Density): For a non-zero signal x G C^“, we define ifs b-densify as follows: 

<5(x) = ||x||i/||x||^. (1) 
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For an all-zero signal x = 07 v^xl^ we define 5(x) = 0. 

Our results in Section |III] show that if A has incoherent columns and the signal x has sufficiently 
low (i-density, then Ax / O^xi- Furthermore, if the columns among A and B are incoherent, then two 
signals x and z satisfying Ax = Bz cannot have low density at the same time. In Section |IVl we show 
that orthogonal matching pursuit (OMP) is able to recover up to 1 /fia non-zero entries of a signal x from 
y = Ax given the coefficients have low density and decay sufficiently fast; here /Tq = maxj^j l 
the coherence parameter of A. This result not only improves upon standard recovery guarantees for OMP 
hy up to 2 X, hut also provides a practical way of recovering the dominant coefficients (in magnitude) of 
signals with low density. We conclude in Section |Vl All proofs are relegated to the appendices. 

B. Notation 

Lowercase and uppercase boldface letters stand for column vectors and matrices, respectively. For 
the matrix M, we denote its adjoint and (left) pseudo-inverse by and = (M^M)“^M^, 
respectively; 1m and OmxN denotes the M x M identity and M x N all-zero matrix, respectively. The 
i-th column of the matrix M is denoted by rrij. Sets are designated by upper-case calligraphic letters; the 
cardinality of the set 5 is |5|. The matrix M 5 is obtained from M by retaining the colunms of M with 
indices in 5; the vector V 5 is obtained analogously from v. We define [x]"'' = max{x,0} for x G M. 

II. Properties of the (5-Density 

The (5-density in Definition [T] enables a finer characterization of important signal properties than the 
signal sparsity ||x||g, which simply counts number of non-zero entries in x. We next summarize the key 
properties of the proposed density measure. 

The following result establishes an upper bound on (5(x); a short proof is given in Appendix lA-Al 

Lemma 1 (5-Density vs. Signal Sparsity): For signals x G C^“, the (5-density in Definition [J satisfies 

0 < (5(x) < ||x||o < Na- (2) 

Equality (5(x) = ||x||g is established if and only if the non-zero entries of x have constant modulus. 

The result in Q implies that if the signal x is sparse, i.e., ||x||q is smaller than its ambient dimension Na, 
then it must also have low density (bounded by ||x||q). Furthermore, for signals whose non-zero entries 
are constant modulus, both the (5-density (5(x) and the signal sparsity ||x||g coincide. In contrast, signals 
with low (5-density must not necessarily be sparse. As an example, consider the following signal class. 


July 13, 2015 


DRAFT 




SUBMITTED 


3 


Definition 2 (a-Decaying Signal): We define an a-decaying signal as a vector x G whose entries 
satisfy Xi = t = 1,..., iVa with 0 < a < 1 — l/Na- 

Fur such signals, the signal sparsity equals the amhient dimension Na as all entries are non-zero; in 
contrast, the (i-density can he hounded hy 

(5(x) < < ||x||o = Na- 

1 — a 

Hence, the (5-density does not simply count the number of nonzero entries, hut also captures crucial 
magnitude information of the signal’s non-zero coefficients—all our results developed in Sections |III] 
and JV] make use of this particular property. 

We now show that the (5-density not only exhibits similarity to the signal sparsity ||x||o, but also some 
fundamental differences. The following result is a consequence of Definition [1] 

Lemma 2 (Inhomogeneity): The (5-density 5(x) and ||x||q are invariant to positive scalings. 

As a consequence, the (5-density and the sparsity ||x||q are not norms. In contrast to the sparsity ||x||g, 
the (5-density also violates the triangle inequality. A short proof is given in Appendix lA-BI 

Lemma 3 (Triangle Inequality): In general, the (5-density does not satisfy the triangle inequality. 

This result has negative consequences for uniqueness proofs that rely on the (5-density. Specifically, a 
common way of establishing uniqueness for sparse signals (see, e.g., ||8l|, |I9]|, ifTTl ) is to consider two 
signals x and x', and investigate Ax = Ax', which implies A(x — x') = Omxi- Thanks to the triangle 
inequality, the sparsity ||x — x'||g can now be bounded by ||x||q -|- ||x'||q; this approach, however does 
not apply to the (5-density, which inhibits the derivation of similar uniqueness guarantees. We therefore 
provide an alternative recovery condition in Section |IV] for OMR 

III. Kernel Result and Uncertainty Relation 

We now develop a novel kernel (or nullspace) result and uncertainty relation for signals with low 
5-density. All our results make use of the following definitions. 

Definition 3 (Dictionary Coherence): Let A G be a dictionary. Then = maxj^j is 

the coherence of the dictionary A. The coherence Hh of a dictionary B G is defined analogously. 

Definition 4 (Mutual Coherence): Let A G and B G be two dictionaries. Then qm = 

maxjj- lafbjl is the mutual coherence between the dictionaries A and B. 

Note that for an orthonormal basis (ONB) A G we have pa = 0. For a pair of ONBs 

A, B G we have the following lower bound: Tm > I/VM (see, e.g., m, 171). 
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A. Kernel Result 

The next lemma is key for establishing our 5-density-based kernel result; the proof is given in 
Appendix IB-AI 

Lemma 4 (Bounds on the Matrix Norm): Let Ag be a dictionary with coherence and 

X G a nonzero signal. Then, the following inequalities hold: 

11 .A.^.A.x 11 

1 -/ia(5(x) - 1) < --2° < 1 + - 1). (3) 

Halloo 

This result resembles lower and upper bounds on the £oo matrix norm, which corresponds to the 
maximum absolute row sum of the Gram matrix A^A. The bounds in ([31), however, depend on the 
5-density of the signal x and enable us to establish the following kernel (or nullspace) result. 

Theorem 5 (Kernel Result): Let A G be a dictionary with non-trivial coherence /Xa / 0 and 

X G be a non-zero signal. If 


5(x) < 1-h 1//Xa, (4) 

then X cannot be in the kernel of A, i.e.. Ax / Omxi- 

Proof: Assume that x / 07 v„xi and x G ker(A). From the left-hand side (LHS) of (l3]l, it follows that 
0 > (1 —/ia(5(x) —l))||x||^ and hence, we have 5(x) > 1 + 1/^la- As a consequence, if 5(x) < 
then X /z ker(A). ■ 

This result implies that non-sparse signals (e.g., with up to Na non-zero entries) having sufficiently 
low density cannot be in the kernel of an incoherent dictionary. To see this, consider an a-decaying 
signal with 0<a<l — (1-|- l/+a)~^- For such signals, we have ||x||q = Na, whereas the 5-density 
based condition in (jU) is always met. Hence, suitably defined densify measures enable us fo esfablish fhe 
imporfanf facl fhaf nol only sufficienlly sparse signals cannof be in fhe kernel of incoherenf dictionaries 
buf also certain non-sparse signals with sufficiently low 5-density. We note that Theorem |5] recovers 
existing nullspace conditions as a special case. From 5(x) < ||x||q we obtain the well-known (and more 
restrictive) condition ||x||q < 1 -|- If+a (see, e.g., lUl, ||5l, ifTTl i. 

B. Uncertainty Relation 

We next provide a 5-density-based uncertainty relation for pairs of dictionaries; the proof is given in 
Appendix IB-BI 

Theorem 6 (Uncertainty Relation): Let x G C^“ and z G be non-zero vectors satisfying Ax = 
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Bz, and let A G g g f^MxNt j^g dictionaries. Then, the following inequality holds 

[l-/ra(<5(x)-l)]+[l-/ife((5(z)-l)]+< 5(x)(5(z)//^. (5) 

This uncertainty relation generalizes the result in HI, 111 for sparse signals to the case of signals 
characterized hy the (5-density. As a consequence of Theorem 0 we have the following result. 

Corollary 7: For a pair of ONBs A and B, the following uncertainty relation holds: l/p^ < 5(x)5(y). 

For maximally-incoherent ONBs (e.g., A being a Hadamard basis and B the identity), we have 
\Xrn = I/'/m 111. In this case, the uncertainty relation in Corollary |7] leads to M < 5(x)5(y), which 
generalizes the well-known uncertainty relation M < ||x||g||y||g by Donoho and Stark |T2|. As a 
consequence, for a pair of incoherent ONBs, a signal with low density in basis A cannot have low- 
density in basis B, and vice versa. Again, we emphasize that sparsity is not the key property that is 
required to establish such uncertainty relations, but rather a suitably-chosen measure of signal density. 

IV. Recovery oe Signals with Low (5-Density 

We next show that signals having sufficiently low density and a fast decaying magnitude profile can 
be recovered perfectly via OMR Our condition guarantees recovery of such signals with up to 2 x more 
non-zero coefficients compared to existing, sparsity-based recovery guarantees. 


A. Orthogonal Matching Pursuit (OMP) 

We now briefly review the OMP algorithm ITTI . IT3l . IT4l . OMP is an iterative method used to recover 
sparse vectors x from y = Ax. After initializing the residual = y and an empty support set = 0, 
OMP selects a column of the dictionary A in every iteration t = I,, Vax according to 




= arg max 




( 6 ) 


Here, the set contains all remaining indices that are not (yet) in the support 

set Then, the index in ® is added to the new support set U and a new 

residual is computed as 


rW = y - A 5 (t)X 5 (t) = (Im - A 5 (t) A^(g)y, (7) 

where x^ct) is the least-squares estimate of the signal’s coefficients on the current support set 5^*^. 
The above iterative procedure is repeated until a (predefined) number of iterations tmax is reached. The 
algorithm’s outputs are the least-squares estimate x^ctmax) and the support-set estimate 
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B. Recovery Guarantee 

We now detail our (5-density-based condition for which OMP is guaranteed to identify the indices 
associated to the largest fmax coefficients in the signal x from y = Ax. Furthermore, if the signal x has 
exactly fmax non-zero entries (i.e., ||x||q = ^max)^ then OMP will recover the signal x perfectly. Concretely, 
we have the following recovery guarantee; the proof is given in Appendix O 

Theorem 8 (5-Density-Based Recovery Guarantee): Assume a non-trivial coherence fia / 0. If the 
maximum number of OMP iterations satisfies 

fmax < 1 “h 1 /ha (8) 

and, in every iteration t = 1,, fmax. the coefficients of the vector x to be recovered satisfy 

(5(x7e(t-i)) < + 1/Fa - {t- 1)), (9) 

then OMP is guaranteed to select an atom associated to (one of) the largest coefficient(s) from the 
vector X 7 ^(t) in iteration t. Furthermore, the support set contains the indices associated to the t = 15^*^ | 
largest (in magnitude) entries in x. 

The first condition dUl ensures that the pseudo-inverse A^(t) in © exists in every iteration. The 
second condition ® imposes a constraint on the (5-density of the signal to be recovered—more precisely, 
it imposes conditions on the magnitude decay of the non-zero entries in x. 

C. Discussion of Theorem |S] 

To make Theorem [ 8 ] more explicit, consider the first iteration of OMP, where we have the empty set 
5 ( 0 ) = 0 and = {!,..., A^a}- In this situation, condition ® requires 

(5(x) < 1(1 -h l/ha) (10) 

to ensure that OMP selects the column associated with a largest entry in xQ In words, if the (5-density 
of the signal x to be recovered is below the RHS of (ITOl ). OMP will identify the index of to the largest 
(in magnitude) entry in x. 

In the second iteration (for t = 2), condition ® requires 

(5(x7^(i)) < 1(1-h l//ra - 1) (11) 

'There could, in general, be multiple entries with the same magnitude; OMP is guaranteed to select one of them. 
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where contains all indices {1,..., except the one selected in the first iteration. While the RHS 
in (fTTl) is more restrictive than (fTOb . the (5-density is now potentially smaller too—this, however, 

depends on the decay profile of fhe magnitudes in x. Rewriting @ reveals that 

d(x7i(t-i)) < C - ^(t - 1) (12) 

with C = ^(1 + 1/fia), ensures that OMP selects the correct atoms in every iteration t (given (l8]l is 
satisfied). Hence, hy successively removing fhe t largest entries in x, we require the (5-density of x^^ct-i) 
to decay at least hy C — — 1). 

We emphasize that for signals whose non-zero entries have constant modulus. Theorem [8] collapses to 
the well-known OMP recovery threshold l|3l, ifTH 

Iloilo < ^(1 + • (13) 

For such signals, we have (5(x7j(t-i)) = ||x||q — (t — 1), which implies that the condition (fT^ is met 
in all OMP iterations (hy setting f^ax = ||x||o)- This observation implies (i) that our condition does 
not contradict existing recovery guarantees for OMP and (ii) that sparse signals with constant modulus 
are worst-case signals from a Theorem [8] and OMP viewpoint. We note that property (ii) is a rather 
well-known fact that can also he observed via numerical simulations. 

We finally show a simple example that demonstrates superiority of Theorem [8] over the standard 
recovery guarantee (fT3] ). Consider again an Q;-decaying signal as in Definition |2] and recall that for such 
signals (5(x) < (1 — o;)”^. If we remove the largest t entries in x, the bound 5{x'ji(,t)) < (1 — a)~^ 
continues to hold. Combining this bound with @, we obtain 

f <2 + l/pa-2(l-a)“^ 

By letting (T —0, corresponding to very fast coefficient decay, we see that one can perform tm&x < l/f^a 
OMP iterations, without violating the conditions ([8]l and (|9l) of Theorem[8] Hence, OMP is able to identify 
the indices associated to the largest f^ax entries (in magnitude), even for non-sparse signals having up 
to Na non-zero entries. By truncating an Q!-decaying signal to the largest fmax < l/f^a entries, OMP is 
able to perfectly recover the signal; this is roughly 2x less restrictive than the standard condition (fT3] ). 

D. Related Results 

We note that Theorem [8] is in the spirit of the OMP recovery condition derived recently in |[T5l Thm. 1]. 
While the condition in ifTSl Thm. 1] imposes a constraint on the decay between consecutive magnitudes 
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in X (ordered in decaying fashion) and requires perfectly sparse vectors that satisfy ||x||g < l//ra- In 
contrast, Theorem [8] makes explicit use of the 5-density and ensures that OMP identifies the largest fmax 
entries, irrespective of whether the signal x to he recovered is perfectly sparse or not. A similar recovery 
result for OMP to recover signals with fast decaying entries has been developed in ifT^ : this result, 
however, relies on the restricted isometry property (RIP), which is a common way of characterizing 
measurement matrices in the field of compressive sensing ifTTl but hard to compute in practice. 

V. Conclusions 

We have developed a novel kernel result and uncertainty relation for signals with low density. Our 
results generalize existing results for sparse signals to non-sparse signals with sufficiently low density. 
Furthermore, we have shown that OMP recovers signals with low density and fast-decaying magnitudes. 
Our results highlight that an appropriately chosen density measure is key for establishing kernel results, 
uncertainty relations, or recovery guarantees, in contrast to results that solely rely on signal sparsity. 

There are many avenues for future work. By modifying the proof of Theorem [H one can obtain a 
recovery condition in presence of bounded measurement noise y = Ax -|- n with ||n II 2 — e using 
techniques developed in HI. In addition, our proofs can easily be extended to block-sparse signals lITSl . 
An analysis of alternative density measures that behave similarly to our 5-density, including 7 (x) = 
||x||2/||x||^ and (t(x) = ||x||^/||x||2 (which was used in, e.g., |[T9l - ll2n i. is left for future work. 
Another interesting direction is the analysis of optimization-based recovery of signals with low density. 
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Appendix A 
Basic Properties 


A. Proof of Lemma [7] 

From Definition [T] we have 

5(x) = = < |Af| = ||x||g, (14) 

maxfc \xk\ ^ \xf\ 

where X is the support set of the signal x and k is the index associated to (one of) the largest coefficient(s) 
of X in terms of magnitude. The inequality in ([14]) follows from the fact that |a;j|/|a:^| < 1, for i £ X. 
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Equality holds if and only if |xi|/|x^| = 1, z G Af, which implies that only signals whose nonzero entries 
have constant modulus satisfy 5(x) = ||x||g. 


B. Proof of Lemma |2] 

The proof follows from a counterexample. Consider two signals x G and z G with entries 
Xi = —and Zi = + e for i = 1,..., Na with 0 < a < 1 and e > 0. Both vectors have the 

following (5-densities: 


No. 


(5(x) = 


1 — a; 

1 — a 


and d(z) = 


1 — a 


Na 


1 T V 1 — o 


+ -^a£ ) ) 


which can he made small hy reducing a. However, the sum of both signals has constant-modulus (with 
magnitude e), i.e., 


(5(x + z) = S(elNaXi) = Na. 

Hence, as (5(x-|-z) > (5(x)-|-5(z), these signals violate the triangle inequality for sufficiently small values 
of a and e, and a suitably large ambient dimension Na- 


Appendix B 

Proof of Kernel Result and Uncertainty Relation 
A. Proof of Lemma |4] 

We start with the lower bound in Q. Since A is a dictionary with = 1, Vz, we obtain the 

following lower bound: 


A^Axll = maxj 


afaiXi + 




> max. 


ijkil - E, 




ai^a,- 


where the last step follows from the reverse and regular triangle inequalities, respectively. Using the 
definition of the coherence we obtain 


A^Axll > 


maxi||xi| - fJ-a\xj\^ 


= max,- 


-|- /Za) Pa j I Xj \ 


X|loo(l + /^a) -fZa||x||i. 


(15) 
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By excluding the case ||x||^ = 0 (implying x / Otv^xi). we finally get the lower hound in Q. The 
upper hound in (l3]l is obtained similarly to (fTSl) and is given hy 

||A^Ax||^ < max*||xi| |xj|} 

< l|x||oo(l -/^a) + fia||x||i. ( 16 ) 

By excluding ||x||j^ = 0, we obtain the upper bound in (O. 


B. Proof of Theorem |6] 

To prove Theorem 0 we will use the following result: 


A^BzIl = 


= max 
i 


af^Bzl = max 


afbfcZfc 


< 


m.sx.y^ Jlm\Zk\ = hmWA 

i f ^ 


k k 

Analogously, we will use ||B^Ax||^ < /rm||x||^. Now, if Ax = Bz, then we have A^Ax = 
and also ||A^Ax||^ = ||A^Bz||^. Using the LHS of Q and (fTTI) . we can bound ||A^Ax|| 
below and ||A^Bz||^ from above, respectively, and obtain 


(17) 

A^Bz 

from 


[1 - ^a(<5(x) - l)]+||x||^ < , 


(18) 


where we take the non-negative part in the LHS because norms are non-zero. Since Ax = Bz, we also 
have ||B^Ax||^ = ||B^Bz||^. Using similar steps as above, we obtain 

[1 - /ib(5(z) - l)]+||z||^ < HmW^Wi . (19) 

Multiplying (fT^ with (fT^ and dividing both sides by ||x||^ INIloo yields the uncertainty relation (|5]l. 


Appendix C 
Proof of Theorem [8] 

A. Preliminaries 

Without loss of generality, we sort the coefficients of x in descending order of their magnitudes \xi\, 
and also sort the corresponding atoms in A; ties are broken arbitrarily. In what follows, we consider the 
resulting “sorted” system y = Ax. 

For OMP to recover the i-th largest atom in iteration i, we need the following condition HI 

max la/^R^yl > max la^R^yl 
i&Ms k£M%\S ' ' 
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to hold. Here, the set 


Ms = {k : max|xj| = \xk\,\/k} 
ieS‘= 

contains all indices corresponding to entries having the same (and largest magnitude), excluding the 
already taken elements in S, and R5 = 1m — A5 is the projector onto the orthogonal complement of 
the columns spanned hy the atoms in the set S previously selected hy OMR Since y = ajXj + A5X5 + 
where 7?. = {1,..., Na}\S, we get the following equivalent condition: 

max laf R5(aiXi + A5X5 + At^uXt^w)! > 
i£Ms 


max |a^R5(aiXi +A5XS +AtjuXt^w)! 
keM%\S 


( 20 ) 


The fact that R5A5 = 0jv^x|5| allows us to simplify (l20l) to 


max afR5(ajXi + A7^\jX7^\j) > max a^R^AT^XT^ . 
ieMs kGM%\S 


( 21 ) 


In order to arrive at a sufficient condition for OMP to select the correct atom in iteration i, we now 
individually lower and upper-hound the LHS and RHS of (|2TI) . respectively. 


max 

ieMs 


max 

i€Ms 


B. Lower Bound on the LHS of (1211) 

We start with the expansion 

af (Im - A5A^)(aiXi + AT^y^XT^y^) 

Xi + afATi\iXTi\i - af As A\;{RiXi + A7^\iX7^\j 

and apply the reverse triangle inequality to obtain the following lower hound: 


max \xi\- af At^wXtjw - af A5A^(aiXi + At^wXt^y 
i£Ms ' ' ' I 


fAsAl 

> max \xi\ (1 + fia) - ha\xk\ - af A^A^afc 

ic- KA c- f ^ ^ ^ 


ieMs 

ken keiz 

We now hound the last term in the above result as follows: 


\Xk\ 


af A^A^afc < ||Af ai||2||(Af A5) ^Afa^ 


.H 


H A aH, 


< 


1^.5 ^»ll2 ll"^.S^fc|| 

[l-/r,(| 5 |-l)]+ - [ 1 -;,„(| 5 |-!)]+• 


< 


hl\S\ 


( 22 ) 
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where we used the Cauchy-Schwarz inequality, Gersgorin’s disc theorem |[22l Thm. 6.1.1], and the 
definition of the mutual coherence jj,m- We note that (1221) requires 


|5| < l//ia + 1, 


(23) 


which must hold for any set size picked hy OMP; as a result, we get condition ([8]l. By combining the 
above bounds, we arrive at 


max |a,^R 5 y| > max IxJ (1 + Un) 
iGMs ' ' i€Ms 


[l-/r.(|5|-l)]+ 


k€Tl 


(24) 


C. Upper Bound on the RHS of (1211) 

We now bound the RHS in (|2T]) from above. To this end, we first expand 




(Im - A^A^)At^xt^ 


af At^xt^ - u^A^A^At^xt^ 


and use the triangle inequality to get 


afR^AT^XT^I < \aj^ATiXTz\ + 


a^ AsA^^Atzxtz 


The first RHS term can be bounded as follows: 


\^k At^Xt^I < ^ ^ /Tal^^fcl ■ 
keK 

The second RHS term can be bounded using the same approach used in (l22l) to get 


i^A^At 


^'RA'R 


< 


dl\ 


[1-^„(|5|-1)]- 


E 

k&V, 


Xk\ 


Combining the above results leads to the following bound: 


max af R^y < 
k&M%\S ' ' 


da + dl 

[l-/r,(|5|-l)]+ 


kGH 


(25) 


D. Combining Both Bounds 

By combining (|2^ and (1251) and using simple algebraic manipulations, we get the following sufficient 
condition for OMP to select (one of) the largest column(s) of A: 


max \xi\> 
i€Ms 


[1 -/ia(|5| - 1)] + 


k£TZ 
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Assuming (|2^ is satisfied, \xi\ / 0, and |5| = f — 1, we finally get the following condition: 




\Xk\ 


ken 


\xi\ 


= ^(X7z), 


which ensures that OMP selects an atom i associated to the largest remaining coefficient Xj (in magnitude). 
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